A  Community  Terrain-Following  Ocean  Modeling  System 

Tal  Ezer 

Program  in  Atmospheric  and  Oceanic  Sciences 
P.O.  Box  CN710,  Sayre  Hall 
Princeton  University 
Princeton,  NJ  08544-0710 

phone:  (609)  258-1318  fax:  (609)  258-2850  email:  ezer@splash.princeton.edu 

Grant#:  N000 1400 10228 

http://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/ 

LONG-TERM  GOALS 

The  long-term  goals  of  this  effort  are  to:  1)  develop  and  evaluate  an  expert  Terrain- following  Ocean 
Modeling  System  (TOMS)  that  will  provide  a  state  of  the  art  model  and  supporting  tools  for 
simulations  of  a  wide  range  of  problems  and  scales;  2)  support  the  Navy’s  efforts  in  improving 
operational  coastal  ocean  forecasting  systems;  and  3)  provide  continuous  support  for  the  terrain¬ 
following  ocean  modeling  community  and  provide  coordination  channels  between  developers,  user 
and  forecasters. 

OBJECTIVES 

The  main  objective  is  to  develop  a  robust  modeling  system  with  various  options  for  relocatable  coastal 
forecasting  systems  for  the  Navy  and  for  other  problems  ranging  from  coastal  to  global  ocean 
circulation.  Such  a  system  should  includes  accurate  and  efficient  numerical  algorithms,  the  latest  suite 
of  vertical  mixing  schemes,  interfaces  for  coupling  with  atmospheric  models,  nesting  capability, 
various  data  assimilation  schemes,  and  a  parallel  framework  for  both  shared-memory  (OpenMP)  and 
distributed-memory  (MPI)  paradigms.  The  system  should  also  includes  web-based  documentation  and 
user-support  software  for  model  set-up,  analysis  and  diagnostics. 

APPROACH 

Developing  and  supporting  community  ocean  models  requires  a  collaborative  effort  by  ocean  model 
developers  and  the  terrain-following  ocean  modeling  community.  This  effort  is  thus  based  on  a  joint 
collaboration  between  the  two  most  widely  used  terrain-following  community  ocean  modeling  groups, 
at  Princeton  University  (T.  Ezer,  PI)  and  at  Rutgers  University  (H.  Arango,  PI).  The  two  Pis,  who 
manage  the  respective  community  models  (POM,  Blumberg  and  Mellor,  1987;  ROMS/TOMS, 
Haidvogel  et  ah,  2000;  Shchepetkin  and  McWilliams,  2003;  Ezer  et  ah,  2002),  play  an  important 
ambassador  role  by  communicating  and  coordinating  with  developers  at  several  other  institutions 
(UCLA,  MIT,  University  of  Colorado  and  others)  and  with  users.  We  also  organize  biennial  users’ 
workshops,  as  well  as  developers  meetings.  The  approach  of  testing  new  schemes  includes 
intercomparisons  of  various  schemes  (e.g.,  turbulence  mixing  options)  and  sensitivity  studies  with 
idealized  and  realistic  cases.  Important  testing  of  new  model  elements  is  done  by  beta  testers  who 
provide  feedback  to  the  developers. 
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WORK  COMPLETED 


Several  new  numerical  features  developed  during  the  past  couple  of  years  have  been  included  in  the 
latest  version  of  the  code  known  as  TOMS/ROMS  2.0;  this  code  was  distributed  to  about  fifty  beta 
testers  in  January,  2003,  and  made  publically  available  to  the  community  in  June,  2003. 

The  code  includes  various  pressure  gradient  algorithms  (Ezer  et  al.,  2002;  Shchepetkin  and 
McWilliams,  2003)  that  are  more  accurate  and  efficient  than  previous  schemes  used  in  SPEM, 

SCRUM  and  POM  (Beckman  and  Haidvogel,  1993;  Chu  and  Fan,  1997;  Song,  1998;  Mellor  et  al., 
1994,  1998).  Several  advection  algorithms  are  available  with  second-,  third-  and  fourth-order  accuracy, 
and  a  predictor-corrector  time  stepping  scheme  may  allow  the  use  of  a  longer  time  step  than  standard 
schemes  do  (for  details,  see  an  evaluation  and  comparison  of  these  algorithms  in  Ezer  et  al.,  2002).  In 
collaboration  with  Warner  and  Sherwood  (USGS),  a  sediment  model  and  generic  length  scale  (GLS) 
turbulence  parameterization  (Umlauf  and  Burchard,  2003)  have  been  added  to  the  code;  this  adds  more 
options  to  the  original  Mellor-Yamada  level  2.5  turbulent  closure  scheme  (Mellor  and  Yamada,  1982) 
and  the  KKP  mixing  scheme  (Large  et  al.,  1994).  A  recent  study  (Warner  et  al.,  2003)  evaluated  the 
performance  of  these  turbulence  closures  in  terms  of  idealized  sediment  transport  applications.  Various 
data  assimilation  schemes,  including  tangent  linear,  adjoint,  variational  data  assimilation  and  ensamble 
forecasting  have  been  developed  and  are  being  tested  through  collaboration  with  scientists  from  the 
University  of  Colorado  (Moore)  and  Scripps  (Miller  and  Cornuelle),  see  Moore  et  al.,  (2003)  and 
Arango  et  al.  (2003)  for  details. 

Particular  model  development  work  at  Princeton  (in  collaboration  with  G.  Mellor  and  others)  includes 
the  testing  of  a  generalized  coordinate  system  (Mellor  at  al.,  2002)  that  allows  to  compare  mixing 
processes  in  z-level  grid  with  sigma  coordinate  or  generalized  terrain-following  grids,  using  otherwise 
identical  numerics  (Ezer  and  Mellor,  2003).  This  work  may  support  the  future  development  of  a 
hybrid,  or  a  general  coordinate  version  of  TOMS.  Other  work  at  Princeton  in  recent  years  focused  on 
improving  turbulence  mixing  schemes  by  including:  1 .  better  parameterizations  of  dissipation  due  to 
internal  waves  (Ezer,  2000;  Mellor,  2001),  and  2.  current-wave  interaction  and  breaking  waves  energy 
contribution  to  both  the  bottom  and  the  surface  mixed  layers  (Mellor,  2002,  2003;  Mellor  and 
Blumberg,  2003). 

As  part  of  the  Pis  role  in  coordinating  the  efforts  of  model  developers  and  in  providing  a  forum  for 
exchange  of  ideas  and  research  within  the  ocean  modeling  community,  two  workshops  were  organized 
during  FY03.  The  second  TOMS  developers  workshop  was  held  in  Boulder,  CO,  July  10-11,  2003, 
with  participation  of  most  model  development  teams  involved  and  ONR  personnel.  The  third  biennial, 
terrain-following  users  workshop  was  held  in  Seattle,  WA,  August  4-6,  2003.  The  meeting  focused  on 
several  important  modeling  issues  such  as  mixing  parameterizations,  data  assimilation  and  new 
developments  in  numerical  techniques  (the  meeting’s  proceedings,  Ezer  et  al.,  2003,  is  available  from 
the  ROMS  or  POM  web  pages).  During  the  last  two  years  both  groups  completed  the  first  stage  in 
improving  the  web-based  information  for  their  respective  community  models 
(www.aos.princeton.edu/WWWPUBLIC/htdocs.pom;  marine.rutgers.edu/po),  both,  in  terms  of  look 
and  services  provided.  Due  to  the  collaboration  between  the  two  groups,  there  are  now  more  common 
diagnostics  and  grid  generation  tools  (matlab  and  netcdf  based)  that  are  being  used  by  users  of 
different  models.  A  new  generic  ocean  modeling  web  page  (www.ocean-modeling.org)  is  partly 
completed;  it  will  serve  the  needs  of  modelers  of  both  groups  as  well  as  the  ocean  modeling 
community  at  large. 
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RESULTS 


The  modular  TOMS  coded  has  been  converted  to  F90/F95,  with  a  parallel  framework  that  allows  both 
shared-  (OpenMP)  and  distributed-memory  (MPI)  paradigms.  Benchmark  tests  are  underway  to 
evaluate  code  performance  on  different  computer  architectures.  For  example,  an  idealized  upwelling 
test  with  small  (128x128x16)  and  larger  (256x256x16)  grid  resolutions  is  used  to  evaluate  and 
compare  OpenMP  versus  MPI  on  the  NOAA/GFDL  Origin  3800  supercomputer  cluster  (Figure  1). 

The  MPI  code  improves  its  efficiency  (up  to  32  nodes)  in  the  larger  grid,  but  its  performance  is 
degraded  due  to  the  increased  communications  cost  over  48  nodes.  On  the  other  hand,  the  OpenMP 
code  exhibits  super-linear  performance  up  to  64  processors  in  the  larger  grid  (Figure  la).  Time 
profiling  over  all  processors  reveals  that,  as  the  number  of  processor  increases,  the  MPI  code  spends 
considerable  more  time  than  the  OpenMP  code  on  output  (Figure  lb).  Serial  I/O  (NetCDF)  is  much 
cheaper  in  shared-memory  configurations.  We  are  currently  working  on  a  parallel  I/O  option  for 
TOMS  based  on  HDF.  Unidata  is  now  working  on  a  parallel  version  of  the  NetCDF  library,  which  will 
be  available  sometime  next  year. 
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Figure  la  (left).  The  speedup  factor  of  TOMS  code  as  a  function  of  the  number  of processors  used. 

Purple  and  yellow  lines  represent  small  and  large  domains,  respectively,  using  the  MPI  code;  the 
cyan  line  represents  large  domain  using  the  OpenMP  code,  which  shows  super  linear  behavior  up  to 
64  processors.  Figure  lb  (right).  The  total  cpu  (sec)  summed  over  all  processors  for  the  large 
domain  MPI  (top)  and  OpenMP  (bottom)  codes.  The  dark  portion  of  each  bar  represents  the  relative 
portion  of  the  calculations  spent  on  writing  output,  which  takes  considerably  more  calculations  in 

the  MPI  code  as  the  number  of  processors  increases. 
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During  the  past  year,  several  projects  at  Princeton  (some  supported  by  other  grants)  contributed  to  the 
general  model  development  efforts.  Studies  of  mixing  processes  in  the  bottom  boundary  layers  (BBL) 
and  turbulence  parameterizations  used  the  generalized  coordinate  system  as  implemented  by  Mellor  et 
al.  (2002),  to  compare  various  grid  classes.  In  particular,  simulations  of  overflow  transports  and 
entrainment  demonstrate  that  a  relatively  coarse  resolution  terrain-following  grid  performs  as  well  as 
much  higher  resolution  z-level  grid  when  all  other  numerical  aspects  are  identical  (Ezer  and  Mellor, 
2003).  The  study  also  indicates  the  difficulty  that  general  coordinate  or  hybrid  models  may  face  in 
implementing  horizontal  and  vertical  mixing  parameterizations.  For  example,  increasing  horizontal 
diffusion  resulted  in  thinner  BBL  in  the  terrain-following  grid,  but  caused  a  thicker  BBL  in  the  z-level 
grid.  These  studies  may  help  in  future  developments  of  hybrid  or  generalized  coordinate  systems  to  be 
evolved  from  a  basic  terrain-following  model  (in  contrast  to  hybrid  models  build  from  isopycnal 
models,  such  as  HYCOM). 

In  recent  years,  attempts  are  being  made  to  couple  ocean  circulation  models  with  wave  models. 
However,  how  to  include  the  surface  waves  motion  and  breaking  waves  in  the  parameterization  of 
oceanic  mixing  is  still  a  new  field  of  research  (Mellor,  2002,  2003;  Mellor  and  Blumberg,  2003). 

These  studies  (as  well  as  other  research  efforts  to  develop  a  theory  for  wave-current  interactions  by 
McWilliams  and  others)  give  a  new  framework  for  improving  turbulence  mixing  in  ocean  models;  this 
efforts  may  be  important  in  particular  for  coastal  forecasting  systems  over  shallow  regions. 

Preliminary  results  (see  above  papers  as  well  as  the  special  session  on  this  issue  in  the  2003  users 
meeting,  Ezer  et  al.,  2003)  indicate  the  potential  of  improving  simulations  of  surface  currents,  surface 
mixed  layers  and  oceanic  thennal  structures. 

IMPACT/APPLICATIONS 

Improved  numerical  schemes  and  new  features  for  terrain-following  ocean  models  is  having  an 
immediate  impact  on  the  many  users  of  this  class  of  models  (over  2000  users  are  now  registered  in  the 
POM  and  TOMS/ROMS  user  groups).  In  particular,  many  users  in  the  Navy’s  labs  and  operational 
centers  presently  use  one  of  these  models. 

TRANSITIONS 

An  official  transition  of  the  latest  TOMS  to  an  operational  Navy’s  code  has  not  been  done  yet,  but 
operational  centers  at  NOAA  and  Navy  labs  has  been  using  either  POM  or  ROMS  being  used  for 
research  and  forecasting  purposes.  These  experiences  and  the  interaction  of  the  Pis  with  the 
operational  centers  will  benefit  future  transitions. 

RELATED  PROJECTS 

Under  separate  funding,  the  Princeton  group  is  involved  in  the  development  and  testing  of  forecasting 
systems  for  the  western  North  Atlantic  and  the  Gulf  of  Mexico  (see  the  Princeton  Regional  Ocean 
Forecast  System,  PROFS,  www.aos.princeton.edu/WWWPUBLIC/PROFS/).  Studies  of  wave-induced 
turbulence  by  G.  Mellor  may  help  to  improve  future  mixing  schemes  in  TOMS.  Our  involvement  in 
the  project,  the  Dynamics  of  Overflow  Mixing  and  Entrainment  (DOME, 

www.rsmas.miami.udu/personal/tamay/DOME/dome.html),  which  recently  became  part  of  the  gravity 
current  entrainment  Climate  Process  Team  (CPT),  will  help  to  evaluate  important  test  cases,  improve 
mixing  parameterizations,  and  compare  results  with  other  models.  There  is  also  ongoing  collaboration 
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at  Princeton  University  with  model  development  efforts  at  the  Geophysical  Fluid  Dynamics 
Laboratory  (NOAA/GFDL). 
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